{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Chapter 12",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jo-cho/advances_in_financial_machine_learning/blob/master/Chapter_12.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WuxJGqNzHl2x",
        "colab_type": "text"
      },
      "source": [
        "There are two ways of backtesting. One is the WF(walk-forawrd) method, and the other is the cross-validation method."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SznquPBBIy9w",
        "colab_type": "text"
      },
      "source": [
        "##WF\n",
        "WF is a historical simulation of how the strategy would have performed in\n",
        "past.\n",
        "\n",
        "Disadvantages\n",
        "- First, a single scenario is tested (the historical\n",
        "path), which can be easily overfit (Bailey et al. [2014]). \n",
        "- Second, WF is not\n",
        "necessarily representative of future performance, as results can be biased by the particular\n",
        "sequence of datapoints.\n",
        "- The third disadvantage of WF is that the initial decisions are made on a smaller\n",
        "portion of the total sample."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i-m9MLeSH9J-",
        "colab_type": "text"
      },
      "source": [
        "## CV\n",
        "\n",
        "The goal of backtesting through cross-validation (CV) is not to derive historically accurate performance, but to infer future performance from a number of out-of-sample scenarios.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wZIMP-z8JJGk",
        "colab_type": "text"
      },
      "source": [
        "Advantages\n",
        "1. The test is not the result of a particular (historical) scenario. In fact, CV\n",
        "tests k alternative scenarios, of which only one corresponds with the historical\n",
        "sequence.\n",
        "2. Every decision is made on sets of equal size. This makes outcomes comparable\n",
        "across periods, in terms of the amount of information used to make those\n",
        "decisions.\n",
        "3. Every observation is part of one and only one testing set. There is no warm-up\n",
        "subset, thereby achieving the longest possible out-of-sample simulation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WgafNAElJYB6",
        "colab_type": "text"
      },
      "source": [
        "Disadvantages\n",
        "1. Like WF, a single backtest path is simulated (although not the historical one).\n",
        "There is one and only one forecast generated per observation.\n",
        "2. CV has no clear historical interpretation. The output does not simulate how the\n",
        "strategy would have performed in the past, but how it may perform in the future\n",
        "under various stress scenarios (a useful result in its own right).\n",
        "3. Because the training set does not trail the testing set, leakage is possible.\n",
        "Extreme care must be taken to avoid leaking testing information into the training\n",
        "set. See Chapter 7 for a discussion on how purging and embargoing can\n",
        "help prevent informational leakage in the context of CV."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TduCqWwJJey-",
        "colab_type": "text"
      },
      "source": [
        "## THE COMBINATORIAL PURGED CROSS-VALIDATION METHOD"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k7xIFGOiKUsQ",
        "colab_type": "text"
      },
      "source": [
        "Given a number 𝜑 of backtest\n",
        "paths targeted by the researcher, CPCV generates the precise number of combinations\n",
        "of training/testing sets needed to generate those paths, while purging training\n",
        "observations that contain leaked information."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZKGSeqWjcN2E",
        "colab_type": "text"
      },
      "source": [
        "CPCV stages\n",
        "1. Partition T observations into N groups without shuffling, where groups\n",
        "n = 1,…,N − 1 are of size ⌊T∕N⌋, and the Nth group is of size T −\n",
        "⌊T∕N⌋ (N − 1).\n",
        "2. Compute all possible training/testing splits, where for each split N − k groups\n",
        "constitute the training set and k groups constitute the testing set.\n",
        "3. For any pair of labels (yi, yj), where yi belongs to the training set and yj belongs\n",
        "to the testing set, apply the PurgedKFold class to purge yi if yi spans over a\n",
        "period used to determine label yj. This class will also apply an embargo, should\n",
        "some testing samples predate some training samples.\n",
        "4. Fit classifiers on the\n",
        "( N\n",
        "N−k\n",
        ")\n",
        "training sets, and produce forecasts on the respective\n",
        "( N\n",
        "N−k\n",
        ")\n",
        "testing sets.\n",
        "5. Compute the 𝜑[N, k] backtest paths. You can calculate one Sharpe ratio from\n",
        "each path, and from that derive the empirical distribution of the strategy’s\n",
        "Sharpe ratio (rather than a single Sharpe ratio, like WF or CV)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ndTF-yz_Kdxr",
        "colab_type": "text"
      },
      "source": [
        "# Exercises"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D49qbx4-Kgps",
        "colab_type": "text"
      },
      "source": [
        "## 1. \n",
        "Q. Suppose that you develop a momentum strategy on a futures contract, where\n",
        "the forecast is based on an AR(1) process. You backtest this strategy using the\n",
        "WF method, and the Sharpe ratio is 1.5. You then repeat the backtest on the\n",
        "reversed series and achieve a Sharpe ratio of –1.5. What would be the mathematical\n",
        "grounds for disregarding the second result, if any?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jeWVzuz6Kmm_",
        "colab_type": "text"
      },
      "source": [
        "A. it is as easy to overfit a\n",
        "walk-forward backtest as to overfit a walk-backward backtest, and the fact that changing\n",
        "the sequence of observations yields inconsistent outcomes is evidence of that\n",
        "overfitting.\n",
        "If proponents of WF were right, we should observe that walk-backwards\n",
        "backtests systematically outperform their walk-forward counterparts. That is not the\n",
        "case, hence the main argument in favor of WF is rather weak."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DraMbyiNMhJX",
        "colab_type": "text"
      },
      "source": [
        "##2.\n",
        "Q. You develop a mean-reverting strategy on a futures contract. Your WF backtest\n",
        "achieves a Sharpe ratio of 1.5. You increase the length of the warm-up period,\n",
        "and the Sharpe ratio drops to 0.7. You go ahead and present only the result with\n",
        "the higher Sharpe ratio, arguing that a strategy with a shorter warm-up is more\n",
        "realistic. Is this selection bias?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8wQLIBLOMq1h",
        "colab_type": "text"
      },
      "source": [
        "A. Even if a warm-up period is set, most of the information\n",
        "is used by only a small portion of the decisions.\n",
        "Although this problem is attenuated\n",
        "by increasing the warm-up period, doing so also reduces the length of the\n",
        "backtest."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UEDnIydGOEKQ",
        "colab_type": "text"
      },
      "source": [
        "##3.\n",
        "Q. Your strategy achieves a Sharpe ratio of 1.5 on a WF backtest, but a Sharpe\n",
        "ratio of 0.7 on a CV backtest. You go ahead and present only the result with\n",
        "the higher Sharpe ratio, arguing that the WF backtest is historically accurate,\n",
        "while the CV backtest is a scenario simulation, or an inferential exercise. Is this\n",
        "selection bias?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F5-yLdWdbbEN",
        "colab_type": "text"
      },
      "source": [
        "## 4.\n",
        "Your strategy produces 100,000 forecasts over time. You would like to derive\n",
        "the CPCV distribution of Sharpe ratios by generating 1,000 paths.What are the\n",
        "possible combinations of parameters (N, k) that will allow you to achieve that?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z8eko5pLOLCo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#path = testing sets = 1000\n",
        "#1000 = k/N(N N-k)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5vWbrA6Se6Vf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from scipy.special import comb"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q-21s4AsjRFe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def path(N,k):\n",
        "  path = k/N*comb(N,N-k)\n",
        "  return path"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SmhHoQo3ia7b",
        "colab_type": "text"
      },
      "source": [
        "n of path = k/N*comb(N,N-K) = 1000\n",
        "\n",
        "n of forecasts = n of path * N = 100000\n",
        "\n",
        "N $\\leq$ 100"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_2qW-ek2jCpl",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "1d38d735-50e4-41a1-fee0-e090d581ca92"
      },
      "source": [
        "for N in range(1,101):\n",
        "  for k in range(1,N):\n",
        "    if 1001 > path(N,k) >= 1000:\n",
        "      print('(N,k)=',(N,k))"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(N,k)= (15, 11)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wGsJ6ozQmZ9S",
        "colab_type": "text"
      },
      "source": [
        "##5.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wuuWEpaplwPn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}